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ABSTRACT 


This thesis research is focused on Network Centric Operations with Unmanned 
Systems (US). It specifically focuses on the currently underdeveloped area of aircraft 
pilot decision support for operating USs, including Unmanned Aerial Vehicles (UAV), 
Unmanned Ground Vehicles (UGV) and Unmanned Surface Vehicles (USV), over the 
network from the board of an aircraft. 

Building on Landreth and Glass’s thesis on controlling UAV over the network, 
including from another manned aircraft, this thesis aims to ease implementation and 
usage of the, SA interface. The SA interface enables the operator to be aware of what is 
going on around the Unmanned System while it is being operated from a remote location, 
and to react in the best possible way within a reasonable amount of time. The Rascal 
UAV interface was reviewed, SA-related problems were identified, and solutions to those 
problems were proposed. After our studies, we proposed eight possible solutions to 
implement, and one of them is implemented and used. However, due to some problems, 
we could not test all our solutions. 
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I. INTRODUCTION 


Chapter I will lay the groundwork for this thesis, including an introduction to 
Situational Awareness (SA), the SA interface and SA views, which are a product of SA 
and the interface. The chapter begins with a discussion of an NPS thesis written by Major 
Kent A. Landreth and Major John C. Glass. Landreth and Glass’s thesis developed the 
concept of controlling Unmanned Aerial Vehicle (UAV) over the network; this thesis 
extends their work by adding a manned aircraft to the network. The chapter continues 
with some definitions of SA from the related literature, an analysis of the useful features 
of those definitions for the purposes of this thesis, and a similar analysis of the definition 
of interface. The chapter concludes with a basic introduction to SA views. 

A. PRIOR THESIS DONE ON CONTROLLING AN UAV FROM AN 

AIRCRAFT 

A thesis done by Major Kent A. Landreth and Major John C. Glass, entitled 
“Extending the Tactical Horizon: Networking Aircraft to Enable Persistent Surveillance 
and Target Development for SOF,” provided the idea behind this thesis. Landreth and 
Glass sought a solution to the Intelligence-Surveillance-Reconnaissance (ISR) needs of 
Special Forces while avoiding Task Priority problems related to ISR capabilities, and 
answering those ISR needs within the Special Forces unit command. They stated that, 
“Our goal is to increase the ISR capacity of units who normally would not rate the 
priority to task a Predator, Global Hawk, or U-2. There are two guiding tenets in 
developing this concept. First, the equipment and its control should be organic to the 
Special Operation Forces (SOF) unit or task force. Second, utilizing this capability will 
not require the soldier to carry any additional equipment into the field.” 

Landreth and Glass point out that a “a gap exists between what is needed by 
special operations personnel to develop potential targets, and what is available to them” 1 


1 Kent A. Landreth and John C. Glass, “Extending the Tactical Horizon: Networking Aircraft to 
Enable Persistent Surveillance and Target Development for SOF,” (Master’s thesis. Naval Postgraduate 
School, 2006), xiii. 


1 



They explain the reason for this gap: “Virtually every asset which can provide FMV or 
even near-real-time imagery over the horizon or Beyond-Line-of-Sight (BLOS) falls into 
the category of Low Density High Demand (LD/HD) systems” and “the demand for the 
ISR products of these assets (U-2, Global Hawk, Predator, etc.) outweighs the limited 
capacity of the relatively low number of these very expensive platforms.” 2 

They developed a candidate solution. “We propose in this paper to introduce a 
concept by which a SOF task force or smaller organization can utilize existing, low-cost 
technologies with Line-of-Sight (LOS) links to obtain a BLOS surveillance capability for 
the purpose of developing potential targets or objectives, without tasking theater-level or 
above assets.” 3 This proposed solution requires that a network of UAV(s) and manned 
aircraft be established to link LOS assets in order to create a BLOS capability. 
Furthermore the real gain and the usability comes from the fact that “this capability can 
be organic to the task force or lower headquarters, thereby increasing that unit’s 
autonomy in launching and tasking its own aircraft, as well as exploiting the resultant 
Image Intelligence (IMINT) or Signal Intelligence (SIGINT).” 4 This solution fills in the 
gap by employing the assets in hand, in a new and more effective way. 

Landreth and Glass developed a basic scenario to test their idea, which involved a 
series of control transfers (of the UAV) between a ground station, a manned aircraft, and 
a forward deployed ground team. According to this scenario, “[t]he UAS will be 
launched manually and control passed to the Ground Control Station (GCS), who will 
then assume aircraft/payload (camera) control via the NPS developed Situational 
Awareness software and server. The Pelican will assume UAS control via SA and recon 
the objective area—simulating BLOS from the GCS—and then send the UAS to a pre¬ 
planned waypoint, where the ground team will assume control through the SA server and 
return the UAS to the objective area for further target development. The Tactical 
Operations Center (TOC) will monitor video and provide scenario updates based on 

2 Landreth and Glass, “Extending the Tactical Horizon: Networking Aircraft to Enable Persistent 
Surveillance and Target Development for SOF,” 3. 

3 Ibid. 

4 Ibid. 


2 



information gathered through mesh video feeds. And the Unmanned Aerial Systems 
(UAS) control will then be returned to the Pelican and GCS respectively for a manual 
landing at the airfield.” 5 6 



Figure 1. Tactical scenario of Landreth and Glass 6 


5 Landreth and Glass, “Extending the Tactical Horizon: Networking Aircraft to Enable Persistent 
Surveillance and Target Development for SOF,” 19. 

6 Ibid. 
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Figure 2. Tactical scenario in a more general form 7 


7 Vladimir N. Dobrokhodov, et al., “New Generation of Rapid Flight Test Prototyping System for 
Small Unmanned Air Vehicles,” paper presented in A!A A Modeling and Simulation Technologies 
Conference and Exhibit , August 20-23, 2007, Hilton Head, South Carolina. 
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Figure 3. Rascal UAV with comm, antennas 8 


8 Landreth and Glass, “Extending the Tactical Horizon: Networking Aircraft to Enable Persistent 
Surveillance and Target Development for SOF.” 
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Figure 4. Aircraft control station 9 


9 Landreth and Glass, “Extending the Tactical Horizon: Networking Aircraft to Enable Persistent 
Surveillance and Target Development for SOF.” 
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Figure 5. Assets of the experiment 10 

In compliance with their scenario, Landreth and Glass developed some objectives 
and measures of performance (MOP) / measures of evaluation (MOE). These objectives 
and (MOP) / (MOE) are as follows: 

B. OBJECTIVES 

• Establish Effective UAS Control via Mesh network (airframe and camera) 
from Airborne and Remote Ground Stations: Also, add UAS waypoints 
and change flight parameters as well as manipulate onboard camera for 
target reconnaissance. 

• Evaluate Video Quality to Determine Aircraft and Sensor Capabilities 

• Use the Mesh Network to Supply Real-Time Video to Pelican, Ground 
Team and TOC for Simulated ISR Target Development 

• Manipulate Mesh Nodes to Simulate BLOS between Parties 


10 Dobrokhodov, et al., “New Generation of Rapid Flight Test Prototyping System for Small 
Unmanned Air Vehicles.” 


7 














C. MEASURES OF PERFORMANCE/ MEASURES OF EVACUATION 

• Safely Transfer UAS and Camera Control Between GCS, Pelican and 
Remote Ground Team 

• Evaluate Image Quality for Target Development and UAS/Sensor Control 

• Functionality of Hardware and Software for Future Development and 
Field Application 11 

In their self-assessment, Fandreth and Glass suggested that they had met the 
objectives either completely or largely. They experienced difficulty in meeting Objective 
Three due to problems of camera stability; but they did meet that objective in theory). 

Fandreth and Glass’s thesis demonstrated some important facts: 

• There is real potential for this concept to yield to an effective means by 
which special operators can significantly increase their ISR capability 
through a low-cost, organic system of networked UAS platforms. 

• Such a network requires no UAS pilot training for SOF personnel, just 
familiarity with whatever mechanism eventually controls the sensor. 

• For the most part, the technology exists today and would not add weight to 
the soldier’s rucksack. 

D. DEFINITION OF SA 

Any decision requires a good information flow to the decision maker. With the 
information from different sources, the decision maker creates a mental picture in his 
head, and the quality and quantity of the information to some point has an effect on the 
completeness and correctness of the mental picture. (However, it should be remembered, 
that good information flow and great SA do not guarantee good decisions, since sound 
decision-making is still needed from the decision maker). The mental picture created can 
be taken as the simple definition for situational (or situation) awareness. Different 
resources off different definitions for various aspects of situational awareness (SA). 
Controlling a system while being aware of the environment plays a major role in different 
areas and professions, and research regarding SA has been done with fighter pilots, chess 
experts, and nuclear power plant operators, Different occupations and areas of interest 

11 Landreth and Glass, “Extending the Tactical Horizon: Networking Aircraft to Enable Persistent 
Surveillance and Target Development for SOF,” 45. 
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may have unique definitions for the given circumstances, however most of the elements 
will be the same due to the characteristics of SA. This section of the thesis looks at some 
of these definitions, and points to important aspects that will be useful in defining SA for 
the pilot of a UAV. 

Dominguez’s definition points to a mental picture of the present, built upon the 
past and used in the future perception of the environment, much like integrating blocks of 
the past and the present for a complete understanding: “Continuous extraction of 
environmental information, integration of this information with previous knowledge to 
form a coherent mental picture, and the use of that picture in directing further perception 
and anticipating future events.” 12 

In the case of Flach’s definition, the dynamic nature of SA is emphasized: “a 
measure of the degree of dynamic coupling between a user and a particular situation.” 13 

Artman and Garbis tie dynamism to the systems making decision, and define SA 
as an intermediate state on the way to the decision: “Situation Awareness (SA) is an 
intermediate state in the decision-making process of dynamic systems where one should 
be able to comprehend the situation in order to make an appropriate decision for future 
development.” 14 

One of the most widely accepted and quoted definitions belong to Mica R. 
Endsley. “Situation awareness is the perception of elements in the environment within a 
volume of time and space, the comprehension of their meaning, and the projection of 
their status in the near future.” 15 


12 Mica R. Endsley and Daniel J. Garland. Situation Awareness Analysis and Measurement , (New 
York: Lawrence Erlbaum Associates, 2000). 227. 

13 “Operator situational awareness and system safety,” 
http://ieeexplore.ieee.Org/Xplore/login.isp7urWiel5/6803/l 8256/00842707. pdf? arnumber=842707 

(accessed January 15, 2008). 

14 Henrik Artman and Christer Garbis, “Situation Awareness as Distributed Cognition,” 
http://www.nada.kth.se/~artman/Articles/Conference/ECCE98wGarbis.pdf (accessed January 24, 2008). 

15 M. R. Endsley, “Toward a theory of situation awareness in dynamic systems,” Human Factors, 
37(1), 32-64, (1995b). 
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In his definition, Endsley emphasizes three components: perception , 
comprehension and projection . Perception refers to recognition of the assets and the 
elements in one’s surroundings and their relative locations and states. It is the basic state 
in the establishment of overall SA. Comprehension is the next state in setting the SA; it 
refers to understanding the meanings of the assets and elements recognized in the 
perception level. For example, what do they mean to a pilot, and what are their effects on 
the pilot’s mission? Comprehension is involved in creating the overall picture for the 
pilot. The last state is projection: what is likely to happen in the near future, who and 
what are going to be where, what are the possible ways that the elements in the picture 
may act 

From the pilot’s view of point, perception is realizing and identifying other 
aircrafts, threats, terrain, and so forth. Comprehension is what these elements mean to the 
pilot and how they might affect his mission. Projection is to consider what the possible 
courses of action are and what entities’ possible courses of action and locations will be in 
the following seconds/minutes/hours. 

A different example might help the reader to understand the terms of SA. In 
football, a quarterback (as well as any player in the field, but especially the quarterback, 
since he can be taken as the decision maker for the team at the time of action) needs to 
identify players on the field (both on his team and the other team), their formation on the 
field, the perimeter of the field, etc.; then he needs to understand the relative importance 
and capabilities of all of the entities (players, ball) he has identified; finally, he must 
consider the possible courses of action of all the entities on the field and how their actions 
would affect his decisions and actions, as well as how his decisions and actions would 
affect their decisions and actions, and how all of this will be affected by where all the 
different entities will be in the future as a result of the possible actions taken. 

Using the definition above, Endsley also identifies three levels of failure 
associated with reduced situation awareness: 


10 



• Level 1 - Failure to correctly perceive information 

• Level 2 - Failure to correctly integrate or comprehend information 

• Level 3 - Failure to project future actions or state of the system. 16 

Any work on SA should aim to eliminate those sources of failure in order to 
enable the operator/pilot’s ability to successfully complete the mission. An interface for 
an SA view is no exception. 

Endsley puts forth another definition that is simpler and more basic. “Situational 
awareness (SA) can be conceived of as the pilot’s internal model of the world around him 
at any point in time.” 17 However, one can argue that this definition lacks the elements of 
past and future states, such as projection, that were contained in Endsley’s prior 
definition. On the other hand, this definition includes the pilot as the subject and, in this 
aspect, applies to the approach taken in this thesis more than the other definitions. 
Furthermore, it can be argued term of internal models points to the same direction of 
Dominguez’s “coherent mental picture.” 

When the definition is viewed from the eyes and the mind of a pilot, some other 
definitions really attract a pilot’s attention due to the terminology they use. One of these 
belongs to Eugene C. Adam. In his work, “Fighter Cockpits of The Future,” Adam 
defines SA quite simply in pilots’ terms: “It’s simply KNOWING WHAT'S GOING ON 
SO YOU CAN FIGURE OUT WHAT TO DO! Where are the friendlies, bogies, SAMs 
and unknowns with respect to my flight? What are their intentions, my intentions and my 
options?” 18 

In his work, Adam looks at SA from the perspective of cockpit design, which is 
similar to the search for a better SA view for the Rascal UAV that this thesis is 

^ Endsley, “Toward a theory of situation awareness in dynamic systems.” 

17 Mica R. Endsley, “Situation awareness global assessment technique (SAGAT),” in Proceedings of 
the IEEE 1988 National Aerospace and Electronics Conference (1988) 

http://ieeexplore.ieee.org/iel2/748/5021/00195097.pdf?tp=&arnumber=195097&isnumber=5021 ) (accessed 

November 20, 2007). 

Eugene C. Adam, “Fighter Cockpits of the Future,” Aerospace and Electronic Systems Magazine, 
IEEE Volume 9, Issue 3, March 1994, 20 - 26, 

http://ieeexplore.ieee.org/iel3/62/6822/00275378.pdf?tp=&amumber=275378&isnumber=6822 (accessed 

November 20, 2007). 
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undertaking. He tries to find the optimum cockpit design to give the pilot the best 
possible SA while looking at his instruments and indicators. He separates SA into two 
parts: “Global and Tactical.” Global - also known as “non visual” - SA has a range of up 
to 200 miles, while Tactical SA consists of the visual range. Adam attempts to find a 
cockpit design that will optimize utilization within the cockpit, and will enable the pilot 
to gain complete SA at both the global and tactical levels. 


SA Has 2 Parts 



Figure 6. Two parts of SA according to Eugene C. Adam 19 

The definition of SA adopted by the U.S. Air Force blends most of the elements 
of these definitions such as dynamism, being continuous, affecting the decision in present 
and future. This definition is as follows: “a pilot's continuous perception of self and 
aircraft in relation to the dynamic environment of flight, threats, and mission and the 
ability to forecast and then execute tasks based on that perception.” 20 One may claim that 
this definition lacks an understanding of past states of surrounding elements, but it can be 
argued that such an understanding is included within the understanding of the dynamic 
nature of the present. 


19 Adam, “Fighter Cockpits of the Future.” 

20 Thomas R. Carretta, David C. Perry, Jr., and Malcolm James Ree, “Prediction of Situational 
Awareness in F-15 Pilots,” The International Journal of Aviation Psychology, 6, no. 1 (1996): 21-41 
http://www.leaonline.com/doi/pdf/10.1207/sl5327108iiapQ601 2 (accessed November 20, 2007). 
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Although there are variations, all the definitions still point in the same general 
direction: that of understanding one’s (the decision maker) position within the 
surrounding environment surrounding in the past and future as well as in the present. This 
understanding gives the decision maker better capability for making decisions. 

E. DEFINITION OF INTERFACE 

For any system, especially any Decision Support System and computer system, 
acceptance by the users depends highly on the user interface. Simply modifying the 
words of Efraim Turban, the key to successful use of any UAV system is the user 
interface. The simpler the use of the system, the greater the chance that it will be utilized 
by the operators. 21 Users cannot judge the features that they cannot use. If an interface is 
not “user friendly,” it does not matter how good the system and features are; the system - 
and the business that makes the system - are most likely destined to fail. 

In his book, “Decision Support Systems in the 21 st Century,” George M. Marakas, 
mentions the user interface as one of the five core components of Decision Support 
Systems (DSS). His five core components are: 

• “Data management systems 

• Model management systems 

• Knowledge engines 

• User interface 

• And the user” 22 

Marakas later defines user interface as a key element in DSS functionality. He 
points out that the data; model and the processing components of DSS must be easily 
accessed and manipulated to support the decision process without hindering it. He also 
states that, ease of use of the interface to communicate with DSS while executing the 
functions mentioned is vital to the success of DSS. 23 

21 Efraim Turban, Decision Support and Expert Systems: Management Support Systems, 3 rd ed. (New 
York: Macmillan, 1998), 223. 

22 George M. Marakas, Decision Support Systems in the 21st Century, (New Jersey: Prentice-Hall, 
1999), 10-11. 

23 Ibid. 


13 




Although the term “interface” may mean an interaction/communication tool, or a 
layer between two or more components (ex. steering wheel, throttle, etc.), or have other 
definitions in different disciplines ranging from chemistry to computer science, 24 the 
definition of interest for this work is that of the user interface in computer sciences, 
especially the Graphic User Interface (GUI). 

The term “GUI” goes back as far as 1945 in theory, but its first practical 
application was achieved by the work of Douglas Engelbart and researchers at the 
Stanford Research Institute in XEROX PARC research and development company in the 
1960s. These studies arguably paved the way for later versions of GUI, even though the 
term would not come into popular use until a bit later. 25 

While an interface is a layer or tool for interaction/communication between 
components in general, in the case of a user interface, one of those components is the 
user. The user is the operator of the system, and the interface provides the tools and 
means that enable the user to operate the system without being distracted by the overhead 
process load. 

Just as there are different definitions of SA, there are different definitions of User 
Interface. Most of those points to interface being the medium between the user and the 
system or unit, relieving the operator of knowing all the complex algorithms and 
command lines by providing manageable symbols for them. Some of these definitions 
follow. 

Marakas defines an interface as “a component of a system that is specifically 
intended to allow the user to access the internal components of that system in a relatively 
easy fashion and without having to know specifically how everything is put together or 
how it works together.” 26 While Marakas points to how an interface saves the user from 


24 Graphical User Interface (GUI), Britannica Online Dictionary, 
http://www.britannica.com/eb/article-9109589/graphical-user-interface (accessed November 28, 2007). 

25 Ibid. 

26 George M. Marakas, Decision Support Systems in the 21 s ' Century, (New Jersey: Prentice-Hall, 
1999), 20. 
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the burden of knowing the internal structure of the system, he carries this further by 
saying, “The easier it is for a user to access the system the better the interface.” 27 

Turban describes the interface as a data transportation medium between user and 
the computer. “The user interface may be thought of as a surface through which data is 
passed back and forth between user and computer.” 28 

According to Turban, GUIs provide visible subjects through which to manipulate 
the system, while hiding the complex syntax from the user in order to simplify the 
manipulation. He says, “GUIs are direct manipulation systems in which the users have 
direct control of visible subjects (such as icons), and actions replace complex command 
syntax. Users just touch or aim at visual areas to interact with a computer.” 29 

Britannica views the GUI as another program within the computer that allows 
communication between the user and the computer: “[a] computer program that enables a 
person to communicate with a computer through the use of symbols, visual metaphors, 
and pointing devices.”30 

Like Britannica, Wordreference online source points to GUI as an interaction tool 
between the user and the computer: “(computer science) a program that controls a display 
for the user (usually on a computer monitor) and that allows the user to interact with the 
system” 31 

Yet another definition from an online source points to how GUI functions in 
terms of operation and data input: “The aspects of a computer system or program which 
can be seen (or heard or otherwise perceived) by the human user, and the commands and 
mechanisms the user uses to control its operation and input data. A graphical user 


27 Marakas, Decision Support Systems in the 21 s ' Century. 

28 Efraim Turban, Decision Support and Expert Systems: Management Support Systems, 3 rd ed. (New 
York: Macmillan, 1998), 224. 

29 Ibid., 233-234. 

3(1 Graphical User Interface (GUI), Britannica online dictionary article 
http://www.britannica.com/eb/article-9109589/graphical-user-interface (accessed November 28, 2007). 

31 Interface, http://www.wordreference.com/definition/interface (accessed November 28, 2007). 
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interface emphasizes the use of pictures for output and a pointing device such as a mouse 
for input and control whereas a command line interface requires the user to type textual 
commands and input at a keyboard and produces a single stream of text as output.” 32 

Merriam-Webster points in the same direction, but also includes some other 
aspects that were missed in the technicality of other definitions such as easiness and 
making choices: “[a] computer program designed to allow a computer user to interact 
easily with the computer typically by making choices from menus or groups of icons” 33 
Even though these two aspects are apparent, they are overlooked in other definitions, and 
this may lead them to be overlooked in design, too. 

For the purposes of this thesis, a combined but rather simple and nontechnical 
definition of interface especially tailored for SA views was developed. According to this 
definition, an interface is a decision support program that enables the user to interact with 
a UAV system and make decisions for its operation. By doing that, an interface enables 
the operator to execute such operations easily and efficiently. 

F. CONVERGENCE OF SA AND THE INTERFACES: SA INTERFACES 

In an UAV system, just like in any system, several components work for the 
overall success of the intended mission. While all of them exist by themselves as separate 
entities, they are put together with the intention of creating a system. The main 
components in a general view are: 

• The UAV itself flying the mission, with all the parts making up the flying 
object: engine, wings, fuselage, control surfaces, landing gear, etc.; 

• Payload for the mission(s), which also defines the nature of the mission(s): 
different type of sensors, armament, exterior payload (medical, food, etc.), 
and radio communication equipment; 

• Control stations (ground station, airborne station, etc.); 

• Takeoff, landing and recovery equipment (if applicable); 

32 Ami Augmented Multi-Party Interaction D.1.2 scenario definition, 
http://209.85.173.104/search?q=cache:-NDPPvZDF2MJ: www.amiproject.org/ami-scientific- 

portal/documentation/annual-reports/pdf/D 1 2.pdf (accessed November 28, 2007). 

33 Merriam-Webster Online Dictionary , http://www.merriam- 
webster.com/dictionary/graphical%20user%20interface (accessed November 28, 2007). 
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• Personnel augmenting the flight of the UAV or controlling the elements; 

• Hardware and software; 

• Last of all, the interface, which transforms all the components into a 
system. 

One can add, subtract or organize the given list of components differently or 
include any of those components within another one. But the main point is that the 
interface is the part which connects all of the components and makes them a meaningful 
system for the user’s intended mission(s), In this case, the user is the operator of the 
UAV. All the capabilities and attributes of the components would not make any sense in 
the overall picture without an interface that enables them to be employed within the 
system effectively. From the SA point of view, a UAV system exists within the 
environment surrounding it, and it is the interface component that enables the operators to 
realize their position and condition within that environment by connecting the elements 
of the system for the user visually and technically. 

In a vastly broad and general look, especially for UAVs, SA interfaces or SA 
views mainly consist of a display of the area(s) of interest(s), a means to display entities 
(including UAV, target(s), enemy and friendly forces, etc.) within the area (mainly by 
icons), and tool boxes either surrounding the area displays, or on separate screens 
designed for that purpose. In general, displays of the area employ a kind of map, 
previously-taken area photo, graphics, or a combination of these. Such displays are also 
known as geospatial displays. 34 Tool boxes surrounding the map are designed to help the 
user execute the mission by giving commands to entities, communicating with the 
entities, changing route, defining entities, etc. Figures 7 and 8 illustrate two of the sample 
SA interfaces (or SA views, as they are generally referred) referenced frequently in this 
thesis. It is possible to observe the components mentioned above in both similar and 
different applications. 


34 Donelly, et al., “Commander’s Intent in User Interfaces for Network-Centric Operations,” (paper 
presented in 12 th ICCRTS, June 19-21, 2007) 

http://www.dodccrp.org/events/12th ICCRTS/CD/html/papers/156.pdf (accessed February 10, 2008). 
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II. BACKGROUND 


This chapter will examine the history of unmanned aviation, as well as the 
concepts of controlling a UAV over the network and from a manned aircraft. The first 
part will focus on the history of UAVs in military forces, especially the U.S. Armed 
Forces, while the second part will focus on the concept and history of controlling UAVs, 
especially over a network. 

A. BACKGROUND ON UAVS AND EMPLOYMENT OF UAVS 

An unmanned vehicle is defined in the US Secretary of Defense’s Unmanned 
Systems Roadmap as “[a] powered vehicle that does not carry a human operator, can be 
operated autonomously or remotely, can be expendable or recoverable, and can carry a 
lethal or nonlethal payload. Ballistic or semi-ballistic vehicles, cruise missiles, artillery 
projectiles, torpedoes, mines, satellites, and unattended sensors (with no form of 
propulsion) are not considered unmanned vehicles. Unmanned vehicles are the primary 
component of unmanned systems.” 35 This definition, however, was not the concept 
utilized throughout the history of Unmanned Systems and UAVs. A number of vehicles 
that would not be considered UAVs according to the above definition will be discussed in 
order to show the progress of Unmanned Aviation. 

The beginnings of unmanned aviation can be tracked back almost to the 
beginnings of manned flight itself, both as an idea and as an operational reality. 
Following the invention of the hot air balloon in 1783, suggestions were made that they 
be used as bombers, by either powering them or by relying on the wind to carry them to 
the enemy’s area of operations. This idea was turned into reality during the siege of 
Venice by the Austrians in the spring of 1849. 36 Although balloon bombers had only 


35 Department of Defense, Office of the Secretary of Defense, Unmanned Systems Roadmap 2007- 
2032, 1. 

3 6 “Remote Piloted Aerial Vehicles: An Anthology, Remote Piloted Aerial Vehicles,” 
www.ctie.monash.edu.au/hargrave/rpav ome.html (accessed March 11, 2008). 
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minor effects on the siege (they were turned back on the Austrians by the wind), this still 
proved that remotely operated (or initiated in this case) vehicles could be used in a battle. 

Even though balloons did not have a big impact in the siege of Venice, their use 
there inspired others to make use of them in later battles. For example, during the 
American Civil War, both sides used balloons loaded with explosive devices as 
unmanned aerial vehicles. The idea was to cause explosions by having the balloons come 
down inside supply or ammunition depots. 37 

Studies about unmanned vehicles did not stop between wars. In the initial phases, 
balloons were studied, largely because they were the option that made the most sense in 
the specific circumstances of the time, given that aircraft had not yet matured and 
wireless technologies were not readily available. During WWI, the US Army worked on 
developing unmanned wind-driven balloons to be used as bomb-carrying attack vehicles. 
Successful program prevented by the armistice to be operational reality. 38 During WWI, 
the US Army also worked on a flying torpedo known as the “Kettering Bug” that was 
inspired by the Curtiss Sperry Aerial Torpedo. 39 The Kettering Bug can be seen as more 
like a cruise missile than a UAV but still, it was an unmanned flight. During about the 
same time period, the British worked on their own unmanned systems, producing a 
device that was able to fly for distances of up to 300 miles. 40 


37 Army communicator, “UAV history,” 

http://www.gordon.army.mil/ac/Summer/Summer%2002/uav2.htm (accessed December 21, 2007). 

38 Charles A. Ziegler, “Weapons development in context: The case of the World War I balloon 
bomber,” 

http://www.istor.org/search/BasicSearch?si=l&hp=25&Search=Search&gw=itx&Querv=Unmanned+air+v 

ehicles+history&wc=on (accessed November 11, 2007). 

39 “Unmanned Aerial Vehicles and Precision Guided Munitions at the Centennial,” 
http://www.uavfomm.com/librarv/defnews.doc (accessed November 11, 2007). 

40 Ibid. 
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Figure 9. Kettering Bug, model on display at the National Museum of the United 
States Air Force in Dayton, Ohio 41 

Between WWI and WWII, studies of unmanned vehicles continued. However, 
around the 1930s, a new area of interest in unmanned air vehicles developed. While work 
continued on the use of unmanned vehicles for bombardment of target areas (both in the 
form of using the vehicles to transport bombs, and of using the vehicles themselves as 
bombs), another branch diverged to use unmanned vehicles as target drones. In today’s 
concept, those vehicles would have to be recoverable in order to be considered unmanned 
aerial vehicles; however, rockets may still be considered to be early forms of unmanned 
vehicles, since they mark a milestone on the way to today’s technology. 

During WWII, UAVs saw action mainly as target drones. Both the United States 
(US) and Japan tried to use balloons and modified manned aircraft (Operation Aphrodite) 
for UAV roles; however, those projects were not successful. 42 Since Joint Publication 1- 
02’s definition states that vehicles should be recoverable, and that ballistic or semi- 
ballistic vehicles, cruise missiles, and artillery projectiles are not considered unmanned 
aerial vehicles, , German V-l and V-2 rockets would not be considered UAVs in today’s 
terms. However, they should be mentioned as a successful step toward unmanned 
vehicles, since they utilized a relatively long distance unmanned flight to reach their 


41 Wikipedia, “Kettering Bug,” http://en.wikipedia.org/wiki/Kettering Bug (accessed February 10, 
2008). 
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Army communicator, “UAV history.” 
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target with a simple form of guidance, and since they were used in a real war with 
relative success. In addition, they were the first example in which unmanned aircraft were 
used, thereby protecting the lives of aircrews. 43 

In between WWII and the Vietnam War, the defined role of UAVs (or RPVs, as 
they were referred to then) was as target drones. This target drone role kept UAV works 
alive for the most part. However, target drones being recoverable led naturally to the 
addition of a reconnaissance role for UAVs. Adding cameras to drones gave 
reconnaissance capability to target drones such as the SD series, starting with the SD-1 
Observer (later designated as MQM-57). 44 

The shooting down of Gary Power’s U-2 on 1 May 1960 might have boosted 
work on UAV reconnaissance development in the 1960s, but anticipation of SR-71 
development and spy satellite programs, as well as President Eisenhower’s commitment 
to end reconnaissance flights over of the Soviet Union, 45 hindered this possibility. The 
Cuban Crisis provided a display of unmanned reconnaissance needs that led to the 
development and use of unmanned platforms in Vietnam. 46 In Vietnam, drones modified 
from their target drone models, such as the Firebee (AQM-34 series), also known as the 
Lightning Bug, were used as UAVs, mainly in reconnaissance roles. Between 20 August 
1964 and 30 April 1975, those UAVs logged 3.435 operational sorties in Southeast 
Asia. 47 


43 “Unmanned Aerial Vehicles and Precision Guided Munitions at the Centennial,'’ 
http://www.uavfomm.com/librarv/defnews.doc (accessed February 10, 2008). 

44 Ibid., and http://www.designation-systems.net/dusrm/m-57.html (accessed February 10, 2008). 

45 Maj. Christopher A. Jones, USAF, “Unmanned Aerial Vehicles (UAVs): An Assessment of 
Historical Operations and Future Possibilities” (research paper presented to the Research Department, Air 
Command and Staff College, March 1997). 

46 Ibid. 

47 Ibid., 10. 
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Figure 10. AQM-34; US Air Force Museum 48 

During the late 1960s and early 1970s, the US Navy studied the possibility of 
converting the Firebee series to a remote-controlled anti-ship missile model, and the US 
Air Force studied the possibility of converting it into an enemy air-defense suppression 
model (with successful air to ground AGM-65 Maverick launches); neither of these led to 
mass production. 49 The end of the Vietnam War also marked the end of interest in 
UAVs.50 



Figure 11. AQM-34 with Maverick AGM 51 

The breakthrough in UAV popularity in modern times began in 1982, after The 
Arab-Israeli war. In his Joint Force Quarterly essay, Ralph Sanders even names UAVs as 
an Israeli military innovation. 52 


48 Directory of US military rockets and missiles, “Teledyn Ryan MQM-34,” http://www.designation- 
systems.net/dusrm/m-34.html (accessed January 3, 2008). 

49 Ibid. 

59 Jones, “Unmanned Aerial Vehicles.” 

51 Ibid.. 30. 

52 Ralph Sanders, “An Israeli Military Innovation,” Joint Force Quarterly Winter 2002-03, No. 33 
http://www.dtic.mil/doctrine/iel/ifq pubs/2033.pdf (accessed February 10, 2008). 
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Employment of UAVs (or Remotely Piloted Vehicles: RPVs) in reconnaissance 
and electronic countermeasure roles marked a milestone in modern warfare and proved 
(or even created) the concept of combat UAVs. “Their role in the suppression of Syrian 
air defense was described as a “wake up call for the US.” 53 

During the 1982 Arab-Israeli war, Syrian SA-6 batteries were tricked by U.S.- 
designed Samson air-launched decoys, which simulated incoming strike aircraft. While 
the SA-6 batteries responded to the decoy attacks, Israel deployed Scout and Mastiff 
RPVs for surveillance and to detect the frequencies employed by the SAM radars. Those 
frequencies detected by the RPVs were jammed by specially configured CH-53 Sea 
Stallion helicopters, while Syrian air defense forces, fooled by the decoys, fell victim to 
Israeli anti-radiation missiles. Massive Israeli air strikes and long-range artillery support 
provided ample firepower, while Israeli RPVs monitored the battlefield. 54 


Figure 12. 
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53 Clifford A. Wright, “The Israeli War Machine in Lebanon in JSTOR,” 
http://links.istor.org/sici?sici=0377- 

919X%28198324%2912%3A2%3C38%3ATIWMIL%3E2.0.CO%3B2-G (accessed January 18, 2008). 

54 Major G.D. Thrash, “USMC, Remotely Piloted Vehicles-The Unexploited Force Multiplier,” by 
Major G.D. Thrash, USMC, CSC 1989, http://www.globalsecurity.org/intell/librarv/reports/1989/TGD.htm 
(accessed January 3, 2008). 
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Figure 13. Scout UAV 55 


Following the operational success of the UAV, interest in them increased; this 
interest was further magnified by the first Gulf War and operations in Kosovo. Learning 
from the Israeli experience, and also motivated by identified needs from conflicts in 
Grenada and Libya, the US Navy started the Pioneer program in the 1980s. Originally an 
Israeli project, the Pioneer program became a joint team effort between Israel and the US 
for the benefit of the US military. UAVs proved themselves valuable ISR assets during 
Operations Desert Shield and Desert Storm. 56 


55 Iweapons, www.israeli-weapons.com , http://www.israeli- 
weapons.com/weapons/aircraft/uav/scout/Scout.html (accessed January 3, 2008). 

56 David Glade, “Unmanned Aerial Vehicles: Implications for Military Operations,” ( July 2000) 
http://www.au.af.mil/au/awc/awcgate/cst/occpprl6.doc (accessed February 10, 2008). 
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Figure 14. Pioneer UAV 57 

This capability, along with added roles such as Combat UAVs, proved the value 
of UAVs in Kosovo, and during the second Gulf War. Successful and impressive 
performances by UAVs in these operations boosted support for UAV employment, 58 and 
made the name UAV an in-house terminology among the military forces of the world as 
well as in the public eye. 

Figure 15, a table from the Unmanned Systems Roadmap, summarizes the history 
and progress of UAVs in the US Armed Forces. 


57 John Pike, “Pioneer Short Range (SR) UAV,” http://www.fas.org/irp/program/collect/pioneer.htm 
(accessed February 10, 2008). 

58 Ibid., 24. 
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I AS Role: 

Brigade division asset for RSTA 


Proponent 

Annv. Marine Corps 


Heritage: 

Falconer (1950-60s) - Aquila (1970-80s) - Pioneer (1980-2000s)-Dragon Drone 
( 1990s) - Outrider (1990s) - Shadow 200 (2000s) 

UAS Role: 

Slupbome asset for reconnaissance and weapon support 


Proponent: 

Navy 


Heritage: 

DASH (1960s) - Project Blaekflv (1970s)- Pioneer (1980-2000*)- Fire Scout 
(2000s) 

UAS Role: 

Small unit asset for over-the-hill reconnaissance 


Proponent: 

Marine Corps 


Heritage 

Bikini (1960s) - Pointer (1980-90s) - Dragon Eve (2000s) 

UAS Role: 

Survivable asset for strategic penetrating reconnaissance 


Proponent 

Annv Air Force Navy 


Heritage: 

Osprev (1960s)-D-21 (1960s) - Classified Program (1980s) - DarkStar (1990s)- 
JUCAS COOOs) 

UAS Role: 

High altitude endurance asset for standoff reconnaissance 


Proponent: 

Air F orce 


Heritage: 

Compass Arrow (1960s) - Compass Dwell (1970s) - Compass Cope (1970s) — 

Condor (1980s) - Global Hawk (1990-2000s) 


Figure 15. Historically Validated UAS Roles 59 


Currently, the US military considers UAVs a better fit for “dull, dirty, or 
dangerous missions than manned aircraft” 60 and justifies this idea by stating, “The 
attributes that make the use of unmanned preferable to manned aircraft ... are, in the case 
of the dull, the better sustained alertness of machines over that of humans and, for the 
dirty and the dangerous, the lower political and human cost if the mission is lost, and 
greater probability that the mission will be successful. Lower downside risk and higher 
confidence in mission success are two strong motivators for continued expansion of 
unmanned aircraft systems.” 61 The roles designated for present and future UAVs in the 
Department of Defense’s most recent Roadmap for UAVs are given in Figure 16. Roles 
ranging from firefighting to air warfare to mine warfare are predicted. Figure 17 shows 
the, roadmap to achieve the vision that is found in UAV Roadmap 2007-2032. 


59 Department of Defense, Office of the Secretary of Defense, Unmanned Systems Roadmap, 2007 - 
2032, http://www.fas.org/irp/program/collect/usroadmap20Q7.pdf (accessed March 12, 2008). 

60 Department of Defense, Office of the Secretary of Defense, Unmanned Aircraft Systems Roadmap, 
2005 - 2030, http://www.fas.org/irp/program/collect/uav roadmap2005.pdf (accessed March 12, 2008). 

61 Ibid. 
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Figure 16. 


Present and Future Roles for DoD Unmanned Systems 62 


6 - Department of Defense, Unmanned Systems Roadmap, 2007-2032, 
http://www.fas.org/irp/program/collect/usroadmap20Q7.pdf (accessed March 12, 2008). 
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Figure 17. Joint Services Roadmap for Achieving DoD Vision for Unmanned 
Systems 63 


As can be observed from these figures, the future for UAVs holds more roles and 
responsibilities than the basic reconnaissance and limited capability airborne gun 
platforms. UAV technologies and concepts can be expected to only mature, and the 
concept studied by Landreth and Glass and the like paves the road for this maturing 
process. 


B. BACKGROUND ON CONTROLLING UAV OVER THE NETWORK AND 

SOME CURRENT EXAMPLES 


Although, balloons are mentioned above as the first unmanned vehicles in basic 
terms, they were not really controllable vehicles, since they were bound by wind 
direction. The roots of successful wireless control can be traced back to 1893, to St. 


63 Department of Defense, Unmanned Systems Roadmap, 2007-2032. 
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Nicola Tesla and his telautomatons (robots); Tesla further demonstrated remote control of 
objects by wireless in an exhibition in 1898 at Madison Square Garden. Then Gabet and 
his 'Torpille Radio-Automatique', by 1909, demonstrated the use of wireless control 
following wireless controlled aircrafts or with more popular and appropriate name R/C 
planes. 64 

In general, UAV control is executed by direct Line-Of-Sight (LOS) or via satellite 
link Beyond-Line-Of-Sight (BLOS). However, network technologies and a special 
branch of networks known as Mobile Ad Hoc Networks (MANET), inspired the idea of 
controlling UAVs over the networks via mobile network nodes. These network nodes can 
be the ground control station launching and recovering the UAV, forward deployed teams 
controlling the UAV for the specified mission in the target area, and the manned 
aircraft(s) controlling the UAV from launch site to target area and flying the mission in 
the target area if needed. 

The technology to establish a MANET and control the UAV over the network 
already largely exists, which enables the military to gain BLOS capability with LOS 
technology. Landreth and Glass’s thesis dealt with the concept of controlling UAVs over 
the network. As discussed in some detail in Chapter I, their thesis proved that the concept 
works. There are also some other studies on the same concept, which one of them will be 
discussed below. 

One study was done under Office of Naval Research-Small Business Technology 
Transfer ONR-STRR program jointly by Nascent Technology Corporation and MIT. This 
study developed a slightly different way of controlling UAVs over the network. Within 
this program, BLOS command of an unmanned helicopter (away helicopter) is achieved 
by using another helicopter (relay helicopter) as a relay to maintain the link between the 
tactical center and the away helicopter. Using a UAV as an airborne relay node is not a 


64 “Remote Piloted Aerial Vehicles: An Anthology, Remote Piloted Aerial Vehicles,” 
http://www.ctie.monash.edu.au/hargrave/rpav home.html (accessed January 5, 2008). 
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new concept; in the case of the Hunter Joint Tactical UAV, the radius of action could be 
increased from 200 km to 300 km by using another Hunter as an airborne relay. 65 

Figure 18 displays actual flight data of the two helicopters during demonstrations 
on August 22, 2005. In the figure, the red circles are the 'away' helicopter, which is 
navigating to the point marked with the red star, and the green circles are the 'relay' 
helicopter. 66 


£ 
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Figure 18. Actual flight data of the two helicopters in the demonstration of ONR- 
STTR 67 



65 David Glade, “Unmanned Aerial Vehicles: Implications for Military Operations,” (July 2000) 
http://www.au.af.mil/au/awc/awcgate/cst/occpprl6.doc (accessed February 10, 2008). 

66 Nascent Technology Corporation, “August 22, 2005: Two helicopters demonstrate beyond line-of- 
sight mission,” http://www.nascent-tech.com/press.htm (accessed January 2008). 

67 Ibid. 
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68 Nascent Technology Corporation, “August 22, 2005: Two helicopters demonstrate beyond line-of- 
sight mission.” 
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III. PROBLEMS EXPERIENCED IN CURRENT AND PAST SA 

INTERFACES 


A. SAMPLE INCIDENTS OF MISHAPS DUE TO SA INTERFACE DESIGN 

FAULTS OR SA LOSS OF THE OPERATOR/PILOT 

This section will discuss some SA and interface related mishaps. These examples 
are from USAF Accident Investigation Board (AIB) Class A report summaries, and from 
personal observations of a TNT experiment on November 2007. These examples point to 
the kind of results that can be created by faulty or deficient interface designs. 

One of the mishaps is taken from USAF accident reports for MQ-1B Predator S/N 
03-3112 on January 17, 2007. This mishap demonstrates interface deficiency in 
continuously monitoring critical engine parameters and displaying engine indication 
deficiencies to the UAV operator. One reason this mishap was selected as an example is 
because the current SA view of the Rascal UAV used in TNT experiments lacks any ki nd 
of alert mechanism, especially any alert regarding Rascal engine parameters such as 
RPM. 


Approximately 14 hours into a 20 hours sortie, the aircraft sustained a 
momentary (two (2) seconds) drop in engine rotations per minute (RPM) 
followed 15 minutes later by catastrophic engine failure. Data logger 
analysis of changes in RPM, oil pressure, turbo oil temperature, and 
propeller pitch subsequent to original two second RPM drop indicate the 
engine was failing over a period of 15 minutes. However, monitored 
engine parameters remained within normal ranges until approximately the 
last minute before the engine seized. Therefore, Mishap Aircraft (MA) did 
not generate any form of caution or warning to the pilot of the impending 
failure until approximately 60 seconds prior to the engine completely 
failing. 69 


69 Executive summary aircraft accident investigation MQ-1B Predator S/N 03-3112. 
http://usaf.aib.law.af.mil/ (accessed January 17, 2007). 
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One other example was observed during TNT experiments at Camp Roberts, CA 
during November 2007. The author of this thesis was observing the operation of a 
“Raven” UAV from the control and operation site at the time of mishap. In the author’s 
judgment, this simple mishap was caused by SA interface-related pilot SA deficiency. 

On November 6, 2007 at 09:09 am, a Raven UAV operated by contractor 
WINTEC flew into a tree while executing its target search and acquisition mission for 
suspected vehicles under an Aerial Search Optimization Model (ASOM) scenario. In the 
case of this specific UAV, basic operation is executed by a team of two personnel: one 
mission controller and one pilot. Due to the SA interface design, the pilot has a limited 
SA since he/she has only the video feed to with which to navigate, and that video feed is 
limited to the camera Field of View (FOV) video feed without an area map aid. Because 
the Raven either completely lacks or has only a limited forward (tilted downward) 
looking camera, the pilot cannot really see obstacles coming at the UAV throughout the 
route. Thus, the pilot is bound to the mission controller for obstacle avoidance in low 
altitude flight. 

During the time of the mishap, the mission controller became distracted when he 
needed to communicate with other UAV personnel flying the Buster UAV in the same 
scenario from the same launch and control site. For future reference, this might be a good 
simulation exercise, since, in real operational conditions out in the field, a mission 
controller and an operator cannot expect to execute their mission in perfectly ideal 
conditions. Distractions should be an expected operational condition in real life, just as it 
happened in this experiment. In this phase of the mission, the Raven was flying from one 
target area to another and no specific event was expected, so both personnel were 
relatively relaxed. While the mission controller was communicating with the other 
personnel, the Raven flew into a tree in ascending terrain in low altitude flight; the pilot 
was not able to see the obstacles in front of the UAVs flight path, and the mission 
controller was engaged with other mission related operations. At the time of the mishap, 
the Raven had been airborne for approximately 75 minutes. The absence of a low altitude 
warning the pilot’s limited SA and his dependence on the mission controller for basic SA 
all contributed to the mishap. 
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The mishap did not affect the experiment, since there were spare Raven UAVs at 
the site. The Raven that was involved in the mishap was recovered shortly with no 
significant damage, and continued flying during the day without any further problems. 



Figure 20. Raven prepares to launch 70 



Figure 21. Buster UAV 71 


70 Directory of US military rockets and missiles, “AeroVironment RQ-11 Raven” 
http://www.designation-systems.net/dusrm/app2/q-l l.html (accessed November 10, 2007). 

7 ' Directory of US military rockets and missiles, “Mission Technologies Buster ,” 
http://www.designation-systems.net/dusrm/app4/buster.html (accessed November 10, 2007). 
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B. UAV INTERFACE PROBLEMS 


This section will briefly examine some of the problems identified with different 
kinds of UAVs and their respective interfaces, with an eye toward identifying some of the 
general areas of dissatisfaction for the users. This will be followed by an examination of 
the interface used by the NPS TNT team for the Rascal UAV. 

Any computer or machine interface may be the source of some sort of user 
dissatisfaction and eventually may lead to an accident or mishap. However, UAV 
interfaces have their unique aspects: 

• Interfaces are created by people with limited UAV control experience: 

• This problem is exaggerated by the fact that the user and the machine are 
located far away from each other, hundreds of miles in some cases; 

• Machine (UAV) feedbacks to the user (pilot, operator) are limited. One 
UAV operator described this by saying, “It’s like driving your car with 
paper towel tubes over your eyes... ; 72 

• The duration and automated nature of the missions create another problem 
area for UAV missions; 

• The relative immaturity of the UAV and combat UAV concepts bring 
some additional problems. 

As was mentioned in Chapter I when SA was defined, decision makers need a 
healthy and correct information flow in order to make healthy decisions. The current 
condition can be compared to a runner running as fast as he can in complete darkness that 
is illuminated only by a very limited and narrow flashlight beam, while simultaneously 
trying to make fast and accurate decisions and trying to identify features and items in the 
dark environment. As soon as the flashlight is turned in a different direction, the 
previously illuminated part turns into complete darkness. In a research experiment on 
ground-based robots, it was found that “search and rescue workers participating in a 
ground-based robot experiment spent, on average, approximately 30% of the time solely 
trying to gain or maintain SA, which chiefly consisted of understanding their remote 

72 Mark H. Draper, “UAV Operator Interface Research at AFRL/HEC,” (May 24, 2004) presented in 
2004, Cerici workshop, 

http://www.cerici.com/documents/2004 workshop/UAVHumanFactorsResearch.pdf (accessed January 13, 
2008). 
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robot’s location, surroundings, and status.” 73 This implies that important time is lost 
while the operator tries to understand the asset’s position instead of executing the 
mission. 

Another problem area results from the duration and highly automated nature of 
the missions, which makes monitoring the UAV flight bothersome at the least. While 
piloting involves feeling the aircraft and seeing the environment surrounding it, UAV 
piloting lacks the classic feelings involved in flying an aircraft: no more feeling the buffer 
on the wingtips, hearing a strange sound, smelling something funny in the aircraft, feeling 
the “G” load, or feel of the pants. 

One more factor increases the potential for problems. The UAV concept is a 
rather new concept in popular use. Moreover, newer usage concepts continue to arise, and 
the designers of the interfaces for those new concepts and the UAVs are generally not the 
actual users in real conditions. It is as if the concept was crawling for years and then 
started to run in a couple of hours, and the body is having a hard time keeping up with 
this sudden change. A decrease in these problems will happen eventually as experience in 
implementation, improved techniques, and better products are developed. This change 
has already begun, as can be seen from the graphs for the Predator UAV, below Figure 
22 . 

When one examines the number of mishaps per flight hour from their first 
inception, the numbers for UAVs look discouragingly bad; for example, “the Air Force's 
RQ-1 Predator accumulated a mishap rate of 32 mishaps per 100,000 flight hours, the 
Navy/Marine's RQ-2 Pioneer 334 mishaps per 100,000 hours, and the Army's RQ-5 
Hunter 55 mishaps per 100,000 hours” 74 But these numbers might not represent the 
whole truth. Even though the number of mishaps increased over the years, the increase in 
number of the hours caused to accident per 100,000 hours ratio to decrease. The ratio of 

73 Drury, et al., “Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface 
Approaches,” http://stinet.dtic.mil/cgi- 

bin/GetTRDoc?AD=ADA456256&Location=U2&doc=GetTRDoc.pdf (accessed February 4, 2008). 

74 William T. Thompson, Stefan H. Constable and Anthony P. Tvaryanas, “U.S. Military Unmanned 
Aerial Vehicle Mishaps: Assessment of the Role of Human Factors Using Human Factors Analysis and 
Classification System (HFACS),” http://stinet.dtic.mil/cgi- 

bin/GetTRDoc?AD=ADA435063&Location=U2&doc=GetTRDoc.pdf (accessed November 10, 2008). 
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mishaps per 100,000 flying hours is actually approaching the manned aircraft level of one 
per 100,000 flying hours. However, interface related mishaps still stand as an important 
factor. According to some sources, more than half of the UAV mishaps are attributed to 
interface designs. 75 

At this point, a closer look at the given figures can help in understanding of the 
trend in UAV mishaps. Figure 22 demonstrates the decrease in mishap numbers as the 
flight hours increase. It should be noted that some of the UAV mishap rates are already 
under the manned F-16 class A mishap rate, and the ones above it are showing decreasing 
trends. 



Figure 22. Class A Mishap Rates (Lifetime), 1986-2006 76 


75 Drury, et al., “Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface 
Approaches.” 

7f) Department of Defense, Unmanned Systems Roadmap, 2007-2032, 
http://www.fas.org/irp/program/collect/usroadmap20Q7.pdf (accessed March 12, 2008). 
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Figure 23 gives a comparative graph of mishaps during initial 50,000 flight hours. 
While F-16 and U-2 followed a widely used and accepted manned aircraft experience, 
UAVs are relatively new entities with new technologies, techniques and implementations 
(including interfaces). Figures 24 through 28 strengthen the view that, as the flight hours 
increase, the mishap rate decreases, which is quite rational since more experience and the 
correction of initial mistakes in design and operating procedures should result in better 
products and procedures. 
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Figure 23. Mishap rate in initial 50,000 hours 77 


77 Bob Nullmeyer and Lt Col Robert Herz, “Predator Decision Making and Human Error: Implications 
for training 21 st Century Warriors,” http://www.cerici.com/documents/2Q07 workshop/4-3-AFRL-Herz.pdf 
(accessed January 13, 2008). 
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78 Bob Nullmeyer and Lt Col Robert Herz, “Predator Decision Making and Human Error: Implications 
for training 21 st Century Warriors.” 

79 Ibid. 
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Figure 27. Predator hours flown 81 


80 Robert T. Nullmeyer, Lt Col Robert Herz, Gregg A. Montijo and Robert Leonik, “Birds of Prey: 
Training Solutions to Human Factors Issues,” http://www.iitsec.org/documents/HP.pdf (accessed January 
13, 2008). 

81 Ibid. 
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Figure 28. Predator Class A Mishap Rates 82 

Another source of problems resulting from the newness of the UAV concept is a 
lack of commonness between user interfaces. Even though there is an effort underway by 
the likes of NATO STANAG 4586 “Standard Interface of the Unmanned Control System 
(UCS) for NATO UAV interoperability” to standardize UAV interfaces, it is as yet far 
from being common. As the armed forces of multiple countries either employ UAVs for 
their missions or plan to develop new UAVs for future missions, as is seen in the 
example of the US Department of Defense’s Unmanned Aircraft Systems Roadmap, they 
realized that there was a possibility of using similar systems within their own structure 
and with foreign forces. Possibly with cost reduction and interoperability in mind, they 
decided to develop NATO STANAG 4586 “to decouple the control station from the air 

82 Bob Nullmeyer and Lt Col Robert Herz, “Predator Decision Making and Human Error: Implications 
for training 21 st Century Warriors,” http://www.cerici.com/documents/2Q07 workshop/4-3-AFRL-Herz.pdf 
(accessed January 13, 2008). 
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vehicle in UAV systems, thus presenting the option of developing a single control station 
to control a range of UAV types in disparate missions and also to allow for greater 
interoperability” 83 Even though it is a step forward, NATO STAN AG 4586 has its own 
problems at this stage, 84 and is not a magical complete solution to Human Computer 
Interface (HCI) problems. 

As Marakas states, as the interfaces became more common, efforts to use them 
and training efforts to learn a system between moving one system to another will 
decrease. 85 So will the costs related to developing a user interface, problems due to 
design deficiencies, and the problems arising from this transformation. 

One of the detailed available user reports comes from the 2001 Global Hawk 
deployment to Australia. In this deployment, Mission Control Element (MCE) operators 
rated status displays and controls in the MCE as consistently unacceptable. The table in 
Figure 29 lists the operator ratings for the Global Hawk UAV. 


83 M.L. Cummings, J.T. Platts, A. Sulmistras, “Human Performance Considerations in the 
Development of Interoperability Standards for UAV Interfaces,” presented at the Moving Autonomy 
Forward Conference, 2006. 

84 Ibid. 

85 Marakas, “Support Systems in the 21 st Century.” 
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Question 

Median Rating 

Rate the mental effort workload during task completion with 
the Global Hawk (GH) UAV system. 

Poor to Acceptable 

Rate the physical workload during task completion with the 

GH UAV system. 

Acceptable 

Rate the time pressure during task completion with the GH 

UAV system. 

Acceptable 

Rate the work backlog during task completion with the GH 

UAV system. 

Acceptable 

Rate the adequacy of tire physical anangement of the 
workspaces in the MCE. 

Unacceptable to Poor 

Rate the lighting in the MCE facility. 

Acceptable 

Rate the heating and cooling facilities in the MCE. 

Poor to Acceptable 

Rate the level of noise in the MCE. 

Acceptable 

Rate the freedom of motion within the MCE 

Poor 

Rate the utility of the status displays and indicators in the 

MCE. 

Unacceptable 

Rate the utility' of the controls and menus in the MCE. 

U nacceptable 

Rate the ease of access to displays, indicators, and controls in 
the MCE 

Unacceptable 


Figure 29. Mission control element MCE Operators’ responses to Human Factors 
Questionnaire 86 

Furthermore, some of the areas are identified as problematic, such as: 

• physical arrangement of the displays (too far apart); 

• the unnecessarily complicated retasking processes; 

• difficult-to-read displays (due to the fonts and colors that were used); 

• difficulty of monitoring the automated system closely over extended 
periods; 

• no access to the vestibular cues that pilots of manned aircraft use to gain 
an understanding of the orientation of the aircraft; 


86 Robyn Hopcroft, Eleanore Burchat and Julian Vince, “Unmanned Aerial Vehicles for Maritime 
Patrol: Human Factors Issues,” http://www.dsto.defence.gov.au/publications/4589/DSTO-GD-0463.pdf 
(accessed January 19, 2008). 
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• no kinesthetic cues that pilots of manned aircraft use to gain an 
understanding of turbulence, weather conditions, aircraft movement and 
gravitational forces. 

Another UAV system and related interface that the author had a chance to observe 
firsthand was the “Bullseye” interface for the Raven UAV operated by WINTECS during 
TNT experimentation at Camp Roberts between November 4 and November 8, 2007 As 
was mentioned before, a relatively simple mishap happened during TNT experimentation 
that yielded some clues about problematic areas with the interface SA relation. The 
operator had limited SA due to the necessity to fly the UAV based only on the onboard 
camera view and real SA is gained through mission controller’s oral directives to the 
operator. Another weak point for the pilot was the lack of any warning about low altitude 
and obstacles on the route. While the mission controller can track the route and way 
points from the map within the interface, if mission controller becomes engaged with 
anything other than the mission, the operator is left with limited SA. 

However, when the two-man team can work together without distraction, they can 
execute the mission with no problem. A feature enabling both the operator and the 
mission controller to see the camera field-of-view (FOV) increases mission effectiveness 
and SA, since they can assess the video feed to map and set their SA easily and quickly 
by recognizing UAV location. This helps them to find and identify terrain features, 
waypoints, and eventually, the target. 
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Figure 30. Screen shot from Bullseye SA view 
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Figure 31. Camera Field of View in Bullseye 


A search of the work on UAV SA views yielded different approaches to SA view 
designs. One interesting design tested an augmented video presentation approach. 87 In 
this specific workshop, augmented and non-augmented approaches were compared. With 
the augmented video display, a screen displays a chase aircraft view which changes 


87 Jill L. Drury, et al., “Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human 
Interface Approaches,” and “Comparing Situation Awareness for Two UAV Interface Approaches,” 
http://www.cerici.com/documents/20Q6 workshop/3-l-abst-MITREcorp-Drury.pdf (accessed 15 January 
2008). 
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attitude in real time as the aircraft flies through the virtual environment. The video feed is 
displayed in the inset box. Researchers geo-referenced the video display to the pre-loaded 
map data, which appears approximately on top of the map area to which it refers. In the 
non-augmented video display, a stationary window of the same size as the augmented 
display is shown. 88 Researchers claimed that with the augmented approach, “operators 
would have a better understanding of where the aircraft was with respect to locations on 
the ground using Augmented Virtuality Interface (AVI) rather than unaugmented 
video.” 89 



88 Drury, et al., “Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface 
Approaches,” and “Comparing Situation Awareness for Two UAV Interface Approaches.” 

89 Ibid. 
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Figure 33. Unaugmented Video interface display 90 


Findings from the experiments favored the augmented display SA view. Results 
are given in the following table. Especially worth mentioning is difference happens at 
users preference of AVI interface to the video interface. (As 2.2/7 to 5.8/7 preference by 
the users would indicate), 

As is apparent from the results in Figure 34, augmenting the video feed to a map 
gives better SA. This may occur because it feels much closer to the feel of flying the 
UAV from a cockpit inside the UAV, thanks to the forward view and the field around it 
being presented realistically. Participants in the research project pointed this out, stating, 
“It’s easier to recognize where the UAV is relative to the entire search space.” 91 


90 Drury, et al., “Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface 
Approaches,” and “Comparing Situation Awareness for Two UAV Interface Approaches.” 

91 Ibid. 
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Result 

Video 

only 

AVI 

Significance 
of difference* 

Average error between marked 
and truth positions for victims 

66 mm 

54 mm 

p < .05 

Likert scale response when asked 
if they had a good understanding 
of the UAV's location with respect 
to the ground 

2.7/7 

3.9/7 

p < .003 

Likert scale response when asked 
if interface helped to perform the 
search task 

3.0/7 

3.9/7 

p < .003 

Likert scale response when asked 
for interface preference 

2.2/7 

5.8/7 

p <.003 


Figure 34. Augmented video design test results 92 


C. SA VIEW FOR RASCAL UAV 

Rascal is controlled via an SA view named “Situational Awareness (SA) Multi 
Agent System,” which was created within NPS specifically for this project by Eugene 
Bourakov. As mentioned in overview document “Situational Awareness (Sa) Multi Agent 
System Overview” for the SA view, “SA Multi Agent System is a program that was 
developed for the purpose of increasing battlespace awareness of ‘war fighters’ and 
combatant commanders during STAN (Surveillance and Target Acquisition Network) 
and TNT (Tactical Network Topology) experiments that are run each quarter at NPS.” 93 
Figures 35 and 36 show the current interface and control elements on the SA view. 


92 Drury, et al., “Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface 
Approaches,” and “Comparing Situation Awareness for Two UAV Interface Approaches.” 

93 Eugene Bourakov, “Situational Awareness (Sa) Multi Agent System Overview,” 
http://cenetix.nps.edu/cenetix/documents/SA overview.pdf (accessed January 25, 2008). 
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Computcr/UAS Interface and Controls 


Designated 
Platform to be 
Controlled 



Figure 35. Rascal SA view interface and controls I 94 


94 Bourakov, “Situational Awareness (Sa) Multi Agent System Overview.” 
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Computer/UAS Interface and Controls II 
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Figure 36. Rascal SA view interface and controls II 95 


In the current version of SA view, users are limited to available maps that can be 
chosen from the list within the view; however, the list of available maps can be increased 
easily to include maps of any operational area. Entities within the area subject to 
operation can be represented with icons on the map, and those are located on the chart in 
latitude-longitude positions. SA agent is a collaborative tool that enables users to 
communicate and realize dynamic occurrences in the area. Users can send text messages 
to each other using the message box tool in the SA agent and dragging and dropping the 
mail icon on the recipient’s icon. 

One of the prime features of the view is that it enables the controller of the UAV 
to manipulate the actions of the UAV from a single screen with relatively simple 
commands. The operator can choose between a video camera and a photo camera and can 
further arrange camera FOV coverage. The ruler feature of the SA view gives the 

9 ^ Bourakov, “Situational Awareness (Sa) Multi Agent System Overview.” 
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capability of measuring between points (Figure 37, points 1, 2, and 3); in addition, it can 
be used to determine the coordinates of points of interest (Figure 37, point 4) on the map 
(which is not possible with another way), by displaying the coordinates in the designated 
box at the bottom left comer of the display screen. 



Figure 37. Ruler function in use 


Figure 38 depicts how to get the information about an entity on the screen. After 
selecting the network conductivity icon from the screen (1), dragging and dropping this 
icon to top of any entity on the screen (2) results in a pop-up information block over the 
screen (3). From this information block, the “Info,” “video,” or “network” buttons can be 
selected for further information and operations. This operation is depicted in sequence 
with the sequence numbers in Figure 38. 
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Another important function, sending and receiving messages, is depicted in Figure 
39. This is especially an important function for the controller on the aircraft because it 
may provide the only communication means to other parties within the network. By 
clicking the sheet icon on top of the “text message window,” the user opens the composer 
window. After writing the message, the user drags and drops the envelope icon on top of 
the entity on the screen. This result in a blinking line between the sender and the receiver 
(on the sender’s screen) as well as a distinctive incoming message sound alert on the 
receiver’s side. This operation is depicted with sequence numbers in Figure 39. 


96 Landreth and Glass, “Extending the Tactical Horizon.’’ 
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Figure 39. Communication via mail 
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Figure 40. Rascal SA view display screen 97 


97 Landreth and Glass, “Extending the Tactical Horizon.’’ 
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Figure 41. Rascal manned aircraft control station 98 


Within all the functions in Rascal’s capability, the most important is probably the 
“still camera” function. As discussed in a recent study, 99 a video feed seems appealing to 
pilots/operators, allowing them to navigate by a continuous image feed similar to the 
normal visual feed to the brain. But from a quality (and quantity) of resolution point of 
view, still picture cameras complement ISR capabilities better. Moreover, Commercial 
Off The Shelf (COTS) photo cameras are more cost effective than video cameras. 

In the case of Rascal, implementation is done by “using a Canon G7 digital 
camera which can be operated wirelessly from an operator on the ground, providing 
focus, zoom, capture and download capabilities, all while the UAS is in flight.” 100 
Installation of the camera to Rascal UAV is shown in Figure 42. 

98 Landreth and Glass, “Extending the Tactical Horizon.” 

99 Kevin Jones, et al., “High Resolution Imaging from a Small Unmanned Air Vehicle,” (November 
2007) http://ieeexplore.ieee.Org/iel5/l 1005/34689/01656418.pdf (accessed February 16, 2008). 

100 Ibid. 
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Figure 42. Installation of Canon G7 camera to Rascal UAV 101 


Without going into all the technical details, in the current version, many camera 
functions are controlled wirelessly via USB. Then pictures taken are stored on the remote 
computer hard-drive in the UAV rather than in the camera’s memory. Image metadata 
enriched by UAV metadata is stored with the picture. An example of metadata stored in a 
typical transmission package is given in Figure 43. As can be seen from the figure, the 
camera generates the first 25 lines of the metadata, and the CCM software adds the rest; 
these are related to aircraft metadata like altitude, speed, coordinates, and so forth at the 
time the shutter-release command is sent. 102 


101 Jones, et at., “High Resolution Imaging from a Small Unmanned Air Vehicle.” 

102 Ibid. 
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Figure 43. Example of Metadata stored in a typical image 


Considering the limited bandwidth between the UAV and the other station(s), 
instead of sending the entire high-resolution picture (which would take minute(s)), a 
thumbnail image is downloaded and displayed in the SA view. Checking the thumbnail, 
the user can decide to download the full-resolution image or not, eliminating unnecessary 
process and bandwidth use. 103 Graphic User Interface (GUI) used for a G7 camera and 
the thumbnail images downloaded from the Rascal UAV can be seen in Figure 44. 
Clicking on those thumbnail images result in a download of the full-resolution images. 

1113 Jones, et al., “High Resolution Imaging from a Small Unmanned Air Vehicle.” 
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Figure 44. GUI for controlling G7 and thumbnail images 104 

With the additional improvements on the software, the capability to automatically 
arrange the timing of the photography in the desired flight path is gained. Furthermore, 
the images taken may be geo-rectified in an overlay with Google Earth. A sample image 
taken during TNT 07-02 is given in Figure 46 as overlaid on a Google Earth image; the 
raw image from Google Earth is shown in Figure 45. 


104 Jones, et al., “High Resolution Imaging from a Small Unmanned Air Vehicle.” 
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Figure 45. Sample raw image from Google Earth 
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Figure 46. Image overlaid in Google Earth 

D. IDENTIFIED AREAS OF PROBLEMS AND REQUIREMENTS FOR THE 
SA INTERFACE 

Interfaces that were examined in this study raised a variety of problems and 
relative solutions. However, those problems and solutions should be evaluated in their 
relative environments and with regard to their mission concepts. 

The defined requirements set for aviation display design of Wickens and 
Hollands 105 will be used to measure the convenience of the user interface, and then to try 
to address some of the shortcomings of the SA view. Wickens and Hollands described 
seven critical principles of aviation display design, which are: 


105 Robyn Hopcroft, Eleanore Burchat and Julian Vince, “Unmanned Aerial Vehicles for Maritime 
Patrol,” http://stinet.dtic.mil/cgi-bin/GetTRDoc?AD=ADA454918&Location=U2&doc=GetTRDoc.pdf 

(accessed January 24, 2008). 
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• Principle of Information Need. Information that is required more 

frequently should be displayed in the most accessible locations; 

• Principle of legibility. Displays must be legible to be useful: That is, large 
enough and with adequate contrast, brightness, illumination, volume (for 
auditory displays), etc.; 

• Principle of display integration/proximity compatibility principle. 
Information sources that need to be integrated or compared should be 
positioned in close physical proximity on the display; 

• Principle of pictorial realism. The display should be a pictorial 

representation of the information that it represents; 

• Principle of the moving part. The moving element on a display should 
correspond to the moving element in the operator’s mental model; 

• Principle of predictive aiding. Predictive information regarding future 
aircraft state is valuable (as long as it is accurate and easily understood), 
and; 

• Principle of discriminability. A display element in a certain context 
should never look (or sound) similar to another element that could occur in 
the same display context. 106 

However, those principles are defined for manned aviation, especially for the 
aircraft cockpit. Even though a UAV interface should relate to cockpit design because 
both are interfaces for the operator of the air vehicle, a UAV interface has its own 
challenges and all the display items for a manned aircraft operator (pilot) may not 
necessarily be needed for the UAV operator. However, those principles may still be 
referred to for determining the necessities for an interface for the purposes of this thesis. 

Another specialty in the UAV case is the level of automation. According to TNT 
faculty member Vladimir Dobrokhodov, UAVs are not really piloted; in a more correct 
sense, the controllers augment the autopilot. 107 This means that every command the pilot 
gives to the air vehicle is in fact a command to the autopilot to command the air vehicle. 
All the unique characteristics of unmanned aviation define a unique set of necessities or 
requirements for the subject of this thesis. 


106 Hopcroft, Burchat, and Vince, “Unmanned Aerial Vehicles for Maritime Patrol.” 

107 Vladimir Dobrokhodov, personal communication. 
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The SA view that is the subject of this thesis is the SA view used for the NPS 
Rascal Unmanned Aerial System (UAS), and, more specifically, that used while flying 
the Rascal from an aircraft, which is the Pelican in this case. It should be remembered 
that the thesis done by Landreth and Glass is a work of concept, and thus ideal conditions 
should not be expected. So, even though it may be used with a more stabilized and unique 
set of conditions in the future, at the moment, this operation is executed via mobile 
platforms, namely laptops. That being said, the same concept can be used by the units in 
an operation area with available flying platforms assigned to military units. As is 
displayed in Figure 47, the conditions of the controller for the specific type of the UAV 
make each case unique. 

The controller is limited to one laptop screen, has limited space, and executes the 
mission in flying conditions. What this means is that, during the mission, the controller is 
subject to the effects of aircraft maneuvers, different light and temperature conditions 
throughout the flight route, turbulence, vibrations, and so forth. So some of the findings 
from different experiments for interface designs may not be applicable to these purposes. 
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Figure 47. Controlling Rascal from Pelican 108 


Tactical Horizon Extension 1—Pelican 

15 Aug 2006 


Some of the shortcomings or problem areas that were identified in conjunction 
with Wickens and Hollands’ principles are as follows. 


1. Problems regarding the SA of the Operator 

Even though the Rascal SA view is not as crippled as the Raven SA view for the 
pilot (in the sense of SA deficiencies from the operator/pilot point of view) and has its 
pros (such as that the video feed display can be moved to anywhere on the screen, and 
resized to provide a more thorough view or occupy less space), it still lacks the complete 
SA gain for the pilot of the UAV in terms of video feed representation and operator SA. 
When video camera feed is displayed in the screen, it comes with a smaller display screen 
on the main display screen, independent from the area on the map that the camera is 
shooting. That is, the video screen location is not necessarily displayed on top of the 

108 Landreth and Glass, “Extending the Tactical Horizon.” 
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actual field of interest that the video camera is shooting. This situation conflicts with the 
“Principle of display integration/proximity compatibility principle, ” because the source 
and the display are not positioned in close or related proximity and the “ Principle of 
pictorial realism ” to some degree, because the display is not realistically displayed in this 
case. 



Figure 48. Video feed display on Rascal SA view 109 

There is nothing in the display to rebuild the pilot’s orientation in the case of 
orientation loss, meaning the loss of the feeling and knowledge of up-down, south-north, 
close-away from a point, etc. When the operator gets busy with something in hand, 
he/she may get too focused on that thing, so that, when the time comes to continue the 
mission for another point or location, he may have lost his knowledge of where the UAV 
is in regard to other entities in the area or to the area itself. Due to such loss of 


199 Dobrokhodov, et at., “New Generation of Rapid Flight Test Prototyping System.” 
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orientation, (which also may be identified as loss of SA, especially perception, in terms 
of answering the “where am I?” question), the operator may lose valuable time in 
operating conditions. 

Another deficiency, which was identified regarding the SA of the operator, is 
related to the perception term of Endley’s SA definition. Current SA does not enable an 
operator to identify his position in compliance with the other entities within the area, or 
vice versa. At any given time, an operator cannot determine distance, heading or time to 
any entity or point within the area. The only tool that can be used in this condition is the 
ruler located on the right side of the screen. As pointed out before when the Rascal SA 
view was introduced, an operator can use the ruler to measure the distance from a point to 
the UAV’s location at the time or to any other point. In Figure 49, this function is 
displayed. The green line between “alert 4” and the point of focus of the photo camera 
shows the line drawn by the ruler, and the value of “1.882 km” shows the distance 
measured between two points. 
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Figure 49. Ruler function usage in current version 


A final deficiency that was identified regarding problems related to the SA of the 
operator is the lack of any reference mechanism for coordination between assets to 
identify a point. This problem was observed during TNT experiments between November 
4 and November 8, 2007, at Camp Roberts CA. While one of the stations using an SA 
view was trying to describe the location of a suspected vehicle to the ground station on 
the van, precious time was lost because of a slight difference between the describing 
party’s intended point and the description understood by the receiving point party. Due to 
this misunderstanding, the UAV flew over the wrong location repeatedly in an effort to 
identify the suspected vehicle. 
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2. 


Problems due to the Nature of Flight Conditions 


Another problem from the aspect of the operator in the aircraft is that the operator 
is subject to the conditions in an aircraft, including the light coming from ever-changing 
angles during flight, throughout the mission. This may cause the operator to lose visual 
sight of the laptop screen, and halt operation. One moment, he/she can have a great 
vision, and seconds later, might lose it completely with a turn. There are also the night 
flight conditions; light coming out from laptop screen can annoy the pilot of the manned 
aircraft. This situation creates a confliction with the “Principle of legibility.” Yet another 
problem due to the same reason is the possibility of giving erratic commands or 
commands that are not finessed over the laptop due to all the vibration and rough/bumpy 
flight conditions that might be encountered during the flight. 

3. Problems due to Lack of Alerts to the Operator 

The last class of problems identified is the lack of alerts to the operator. At the 
current state of operation, the operator in the aircraft has no means of getting an 
indication of possible problems. The only time he/she is going to learn about a problem is 
when it happens, be it engine malfunction, connection loss with other stations or the 
UAV, closure to restricted zone, etc. In addition, any potentially dangerous situation l ik e 
low altitude flying can go unnoticed due to the lack of an alert mechanism. In conjunction 
with the flight conditions, incoming messages may remain unnoticed, since the operator 
may have a hard time hearing the email received sound alert. These problems are in 
conflict with the principles of “ Principle of display integration/proximity compatibility 
principle” and “ Principle of information need. ” 
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IV. FINDINGS AND SUGGESTIONS FOR AN AIRCRAFT PILOT 
SITUATIONAL AWARENESS INTERFACE FOR AIRBORNE 

OPERATION 


Based on this researcher’s experience as an operational pilot, and observations 
from the review of the related literature, some suggestions for the Rascal interface were 
developed. 

A. POSSIBLE SOLUTIONS FOR PROBLEMS FOUND 

While searching for solutions to identified problem areas, the special situations 
related to the Rascal UAV, its mission conditions, and its SA view were considered. The 
current view offers flexibility, low cost and ease of implementation, all of which should 
be retained. Using a laptop (or even a smaller medium like a PDA) for the control of the 
UAV enables the operators to operate from different kinds of platforms, since they are 
not tied completely into a specific platform. Furthermore, there is, in reality, only so 
much bandwidth and computing power available at a reasonable cost. 

1. Problems regarding the SA of the Operator 

While examining the current literature about SA displays, one SA view design 
stood out; it is depicted in Figure 50. In this view, the UAV is depicted from the chase 
aircraft view and the video feed is embedded on top of the display map at the actual 
location the video is shooting. This technique mitigates the “looking through soda straw” 
feeling, because the operator of the UAV actually can sense the area other than where the 
video is shooting, and can have better SA since he/she can evaluate the UAV’s location 
and the environment surrounding the UAV. Also, the operator does not need to move 
his/her eyes through different locations over the display to create a meaningful picture of 
the area in the mind and match the video feed to the area. One other capability gained 
with the design is a chase aircraft view. A chase aircraft view creates a more inside the 
UAV feel than the current view of the UAV from sky (or god’s eye/bird’s eye view). 
However, this display comes with its own challenges. There is always added computation 
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power need with such a design and the laptop is limited in this regard at the moment. In 
addition, there is a bandwidth problem and this kind of design would require additional 
bandwidth to keep the steady video feed to stations from the UAV. Another limitation 
due to use of a laptop is the fact of a limited screen. Depending on the altitude at which 
the mission is executed, the video feed can be so small on the map that identifying the 
objects on the video feed can be really hard, or it may be so big that it may cover a big 
part of the screen to the point that the entire gained SA might be lost. Even the focus 
feature of the camera may not be helpful. Also, a chase aircraft view requires that a view 
of the map be angled to match the video feed and chase view in order to be realistic. 

Based on these considerations, the resizable and not embedded video feed screen 
display of the current design for the Rascal UAV is favored. At this point, it is important 
to remember that the maps of the areas are preloaded on the SA view, and embedding the 
video feed as in Figure 50 can be a technical challenge. An attempt was made to come up 
with suggestions that offered minimal added complexity over the current design. 



Figure 50. Embedded video feed to display screen 110 


110 Drury, et al., “Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human 
Interface Approaches,” and “Comparing Situation Awareness for Two UAV Interface Approaches.” 
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To address these points, another design example encountered while researching 
this project was utilized: the “Bullseye” SA view of the Raven UAV operated by the 
WINTEC. The appealing feature of the Bullseye design is a rather simple addition that 
creates added SA according to mission controller and operator involved in TNT 
experiments at Camp Roberts on November 6, 2007. This simple addition is a trapezoid 
representing the camera FOV coverage on the map, shown inside the red circle in Figure 
51. Although one may think it is already apparent where the camera is shooting, in 
reality, knowing the approximate borders of the camera coverage makes matching the 
video feed to the area easier. 



Figure 51. Bullseye SA view for Raven UAV by WINTEC 
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The suggested design for this problem is depicted in Figure 52. With the basic 
addition of the pictural display of camera FOV coverage on the map, the area-video feed 
match is improved. This feature is already present with the current SA view for the photo 
camera view and can be arrasnged according to altitude and zoom level as shown in 
Figures 53 and 54. 



Figure 52. Suggested SA view VI (version 1) 
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Figure 53. Before altitude and FOV degree change 
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Figure 54. After altitude and FOV degree change 


A simple north arrow or a compass rose on the map can help the operator in 
orienting himself/herself to area. Also, giving the heading and distance to a point or entity 
from the UAV by clicking on the entity would enhance this orientation. 

As was discussed earlier, in the current SA view, maps are preloaded and fixed 
for the area of interests. As a result, there are no navigation options like ‘north up” 
(which orients the map to show north on the upper side all the time) or “route up” (which 
orients the map to show the route direction all the time). The proposed modification to 
the SA view should help overcome this issue. Since the area maps are fixed and 
preloaded for the mission area, adding a symbol pointing to north is easier and does not 
have to change over the map, because the user can’t change the area map orientation on 
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the screen. So a basic north arrow or compass rose should be added to the top right part 
of the screen; these can be relocated in order not to hinder the operations by blocking the 
screen. 
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Figure 55. Suggested view with north arrow or compass rose 


When it comes to displaying distance and heading to an entity, two different 
approaches were considered: first, adding a feature that would display distance and 
heading to any point or entity, when this entity or point is clicked; and second, 
embedding a compass rose over the UAV icon (or any entity in this matter) and using the 
ruler function to read the distance while reading the heading manually over the compass 
rose. However the second version would not bring a significant new capability to the 
table, since the operator still needs to use the ruler to get the distance. Given the current 
flight conditions in the Pelican, it is preferable to minimize manual commands. 
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Figure 56. Suggested view with distance heading function version 1 
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Figure 57. Suggested view with distance heading function version 2 


Finally, for the problem of a lack of a common reference mechanism between 
participating parties, the simple yet rather effective mechanism of implementing a grid 
mechanism over the area map is suggested. 

With the current view, creativity in using the existing features makes a partial 
solution possible. This solution can be possible by using “Alert” icons on the left side of 
the screen. The party describing a point can locate one of the (predetermined) alert icons 
on the intended point on the map, and the other parties can then identify the location. 
However, if the alert icons are already in use, adding another alert icon on the map may 
cause confusion between participating parties. One of the parties may confuse one of the 
already-existing icons for the newly-added icon identifying the marked location. (Even 
though icons are in different colors, they have the same shapes and patterns). 
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Due to the possibility of such confusion, another solution is suggested: Adding a 
grid view over the map. This grid view would activate when the “grid” button is selected 
(causing the button to turn green), and deactivate when the button is reselected (causing 
the button to turn red). A simple depiction of this suggestion is given in Figure 58, below. 



Figure 58. SA view with grid imposed over the map 


2. Problems due to the Nature of Flight Conditions 

There are also problems that arise from the fact that the controller is located on an 
aircraft. Again by staying within the limits described earlier, some simple but probably 
effective solutions for those ki nds of problems are suggested. The main problem that was 
identified was that light coming from different directions throughout the mission can 
reflect at different angles and can make the screen unreadable at times. A simple foldable 
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hood approach can solve this problem for the most part while staying within this project’s 
limits. This approach was also used by the WINTEC team for the operator of the Raven 
UAV. 

Yet another problem identified under this category was difficulty in putting the 
commands on the laptop finger pad while experiencing vibrations, aircraft maneuvers, 
and turbulence. A mouse pad and wireless mouse were considered in an attempt to 
mitigate this problem, but they also had downsides like lack of space and additional gear 
required to keep them under control. Eugene Bourakov suggested a better idea: a finger 
roll type mouse. 

This mouse, suggested for use in an airborne control platform due to limited space 
in Pelican cockpit, is shown in Figure 59. The scroll ball (track ball) on the mouse 
enables the operator to move a pointer in the screen, while two buttons on top serve as the 
“right and left click buttons” of the standard, widely-used mouse design. The trigger type 
button in the bottom enables the operator to use the right click function of the standard 
mouse while moving the pointer on the screen with the scroll ball. This mouse design 
helps the operator in the aircraft to give commands, send messages, etc., without the 
limits (such as the need for a flat surface, etc.) of the standard mouse design. 
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Figure 59. Finger roll type mouse 


3. Problems due to Lack of Alerts to the Operator 

As previously mentioned, there is no alert mechanism implied on the SA view, 
and any potential problems may go unnoticed until they actually happen. 

In the current situation, the ground station gets the information regarding engine 
parameters, communication links, and the health of the UAV. But those are not 
transferred to the operator on the aircraft. Also, some other important parameters, like 
low altitude or slow speed, which may indicate upcoming dangers, go unnoticed because 
there are no means to engage operator’s attention. 

One other important point that may go unnoticed is incoming messages. As 
mentioned earlier, messages can become the only means of communication between 
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stations and may be vital to mission success. In the current version, a new message 
creates a sound alert on the receiver’s screen. However, this alert may go unnoticed due 
to the noise level in the aircraft and the workload of the operator. 

Most of the UAV operations including Rascal, are largely automated flights; 
when the operator intervenes to control the UAV, his commands are actually commands 
to the autopilot to command the UAV. Automation in aviation can lead to some problems 
like a requirement for increased monitoring/watch keeping, failure to monitor progress of 
the flight, and minor input errors with serious consequences. 111 An alert mechanism 
plays the role of circuit breaker in these conditions, by letting the operator know when 
things are going wrong that might go unnoticed otherwise. 

Our suggestion to deal with the lack of an alert mechanism was to create a caution 
mechanism on the SA view similar to the ones in aircraft. A “Master Caution” alert on 
the screen is suggested to visually catch the attention of the operator. This “Master 
Caution” alert should appear in the middle of the screen in a red blinking box. At the 
same time, this box should be transparent to some level, in order not to halt the 
continuing mission while serving its purpose. In compliance with the “Master Caution” 
alert in the middle of the screen, there should also be a Master Caution panel, with a 
blinking red box for the respective malfunction. Any malfunction or dangerous situation 
should activate a “Master Caution” alert and respective “Caution” light. This design 
suggestion is depicted in Figure 60. Thresholds for those respective alerts should be set 
either permanently or per mission (engine speed, altitude, etc.). How those alerts will be 
evoked is the subject of another discussion. 

For incoming messages, a similar approach with slight variations is 
recommended. For incoming messages, a yellow blinking “New Message” ellipse should 
appear at the top middle of the screen. This suggested design is also depicted in Figure 
60. These variations in location shape and color would prevent confusion between the 
two master alerts. This measure also complies with the “Principle of discriminability. ” 

111 Anthony Ciavarelli, “Cockpit Control and Display Design Hazard Analysis,” 
https://www.cnet.navy.mil/nascweb/sas/files/cockpit control and display hazard checklist.pdf (accessed 

March 5, 2008). 
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Figure 60. Recommended SA view design before meeting 


After candidate additions to SA view for Rascal were developed, Dr. Bourakov 
was consulted for further refinement of the design. Dr. Bourakov suggested that the 
mission screen should be kept less crowded with additional icons and alerts, since the 
screen has limited space and blocking this available space might hinder the operation. 
Instead of putting alerts on the main screen, he recommended putting them on the left 
side of the screen. A blinking message box when a new message is received would take 
the place of a “new message” icon in the main screen. For the “master caution” box, he 
recommended that the blinking master caution box be put on top of the message box, 
which, when clicked, would activate an additional master caution panel with the related 
information about the cause of the “master caution.” These additions would comply with 
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Dr. Chiavarelli’s notion of warning signals being adequate to gain the attention of the 
crew 112 while invading less space in the limited screen. 

Applying his suggestions to the next version of the SA view resulted in the SA 
screen depicted in Figure 61. According to this screen, any condition triggering a Master 
Caution light blinking action would lead the operator to click on the master caution (1) 
icon. Clicking on the master caution light would cause a “Warning Panel” to come up on 
the screen (2). On this panel, the conditions that activated the master caution would be 
displayed with different color codes depending on their urgency (more important ones in 
red, others in yellow). If the operator wants to get more detailed information, clicking the 
“Details” button next to the threat would result in a “Warning Detail Panel” opening up 
on the screen. This details can give the time of detection of the threat, current state, worst 
condition, and the related times. It is important that alert information be prioritized, since, 
even in a condition where all the necessary information is depicted, operator(s) can 
overlook vital information while focusing on less important information. As research 
shows, 35% of SA errors involved situations where all the needed information was 
present, but was not attended to by the operator. 113 

All of the panels that have opened on the screen can be relocated by clicking and 
dragging over the screen, and can be resized so as not to hinder ongoing operations. 


112 Ciavarelli, “Cockpit Control and Display Design Hazard Analysis.” 

113 Mica R. Endsley and Daniel J. Garland, Situation Awareness Analysis and Measurement, (New 
York: Lawrence Erlbaum Associates, 2000). 
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Figure 61. Recommended SA view design Version 2 (V2) 
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V. SAMPLE INTERFACE DESIGN 


A. PROPOSED FEATURES FOR THE SA VIEW 


This chapter summarizes the proposed SA view design with all the new additions 
displayed. The SA view before the proposed changes are presented one more time to 
make the comparison easier for the reader. All the proposed solutions are intended to be 
cheap and easy to implement. The proposed changes are also designed to preserve the 
flexibility of the current SA interface, since this feature is unique and important for user 
preference, ease of use, and the eventual success of the SA view. 


Figure 62 depicts the current SA view. Figure 63 displays the proposed additions 
to the current SA view: (1) Master Caution panel, (2) Flashing Message box in case of 
message received, (3) Camera Field of View Indicator, (4) Grid Button, (5) Distance 
Heading Button, and (6) North Arrow. 
















Figure 63. SA interface with proposed features added 


The following figures display the features added in use. Figure 64 displays the 
Master Caution panel. This feature fills an important hole in the current design from the 
point of the UAV operator in the manned aircraft. These cautions will enable the operator 
to take necessary steps instead of watching helplessly when emergencies happen. The 
types of cautions and thresholds should be flexible in order to be able to change 
according to mission and UAV types and requirements. This work does not suggest 
certain types and thresholds. However, at a minimum, low RPM, low power and low 
altitude cautions, as well as link health cautions should be included. 
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Figure 64. Master Caution Feature 


Another feature is displayed in Figure 65. Grid implementation is a useful 
reference tool for coordination of the efforts of the parties participating in the mission. 
This feature should minimize confusion between the parties in their efforts to identify 
target locations, as well as defining sectors or areas on the map. 
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Figure 65. Grid Feature 


Another proposed feature is a bearing/heading feature and north arrow on the 
map. Both features will minimize the time for the pilot to orient himself/herself on the 
field while navigating between points or searching for targets. They will also enable the 
operator to understand his position in relation to other entities in the field. 
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Figure 66. Distance Heading Feature and North Arrow Feature 


B. DESIGN TESTING AND EVALUATION 

When this thesis was begun, the plan was to identify the weaknesses and missing 
features in the current SA interface, find the appropriate solutions to those weaknesses 
and propose further solutions for the missing ones. After developing applicable solutions, 
the plan was to implement those modifications to the SA view, and then test them in the 
field during the TNT experiment. 

However, both a relatively short timeline, and the busy schedules of the thesis 
advisors limited the implemented changes. Furthermore, during the TNT experiment 
from February 25 to 29, it was learned that the SA view usage for the control of the 
Rascal UAV had changed radically. The initial plan was to use the SA view from the 
Pelican manned aircraft during the TNT experiment. But the configuration of the Pelican 
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for this experiment allowed only the pilot to be present in the aircraft during the flight, 
which eliminated the chance to fly and test the interface. A secondary plan was to use the 
interface in a truck during the TNT experiment, but that did not go as planned either, 
since it was determined that that would not add any value to the testing process. The last 
plan was to use the interface during the final day of the TNT experiment, but this didn’t 
go as planned either, since control of the UAV was to be executed by the team in the van 
using a Unix-based interface and all that could be done in the Tactical Operational Center 
(TOC) was to observe the process and message using the current interface. 

All those factors eliminate the chance to evaluate the SA view changes. But at 
least one of the proposed changes was implemented and used. Eugene Bourakov changed 
the Message Box to flash when a message is received. Even though a formal evaluation 
process was not executed for this added feature, the initial user response was positive. In 
addition, a finger roll-type mouse is easy to implement and it is expected that it can be 
used in similar conditions, regardless of the implementation. 

The other proposed additions should create a SA increase with relatively low cost. 
However further testing is necessary in order to test these ideas. The features proposed 
here can be used as references for future SA view designs for the NPS Rascal UAV, and 
will hopefully make the operators’ job easier by arming him/her with better SA. 
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VI. CONCLUSION SUGGESTIONS 


This research was undertaken with the goal of improving the SA of the UAV 
operators, in this particular case, the operators of the Rascal UAV used by the NPS TNT 
project. By conducting a literature review of SA and interfaces, followed by the history of 
UAV and control over the network, the shortcomings of the SA view currently used for 
the Rascal UAV were identified in order to provide the operator with good SA and ease 
of use for operational missions. 

The specific problem areas identified during the research in this phase were 
classified our findings into three categories: 

• Problems regarding the SA of the operator; 

• Problems due to nature of flight conditions ; 

• Problems due to lack of alerts to the operator. 

Classifying the problem areas was followed by developing candidate solutions for 
those problematic areas. The proposed solutions aimed to be easy and relatively cheap, 
both in terms of money and in terms of time, to implement. The proposed solutions are: 

• Use of an FOV frame on the map; 

• Use of a north arrow; 

• Use of a grid over the map; 

• Use of a hood for changing light conditions; 

• Use of a finger roll mouse; 

• Use of a flashing message box for incoming messages; 

• Use of a “Master Caution” light and panel; 

Unfortunately, there was no chance to test these proposed solutions or see all of 
them implemented; however, they should provide guidelines for addressing deficiencies 
in current SA views and developing pilot SA interface for future analyze. 
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